

<!DOCTYPE html>
<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
<head>
  <meta charset="utf-8">
  
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  
  <title>Tox21 Challenge &mdash; OpenChem 0.1 documentation</title>
  

  
  
  
  

  
  <script type="text/javascript" src="../_static/js/modernizr.min.js"></script>
  
    
      <script type="text/javascript" id="documentation_options" data-url_root="../" src="../_static/documentation_options.js"></script>
        <script src="../_static/jquery.js"></script>
        <script src="../_static/underscore.js"></script>
        <script src="../_static/doctools.js"></script>
        <script src="../_static/language_data.js"></script>
        <script async="async" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.7/latest.js?config=TeX-AMS-MML_HTMLorMML"></script>
    
    <script type="text/javascript" src="../_static/js/theme.js"></script>

    

  
  <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
  <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
    <link rel="index" title="Index" href="../genindex.html" />
    <link rel="search" title="Search" href="../search.html" />
    <link rel="next" title="API documentation" href="../api-docs/blocks.html" />
    <link rel="prev" title="GraphCNN for predicting logP" href="gcnn_tutorial.html" /> 
</head>

<body class="wy-body-for-nav">

   
  <div class="wy-grid-for-nav">
    
    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
      <div class="wy-side-scroll">
        <div class="wy-side-nav-search" >
          

          
            <a href="../index.html">
          

          
            
            <img src="../_static/logo.png" class="logo" alt="Logo"/>
          
          </a>

          

          
<div role="search">
  <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
    <input type="text" name="q" placeholder="Search docs" />
    <input type="hidden" name="check_keywords" value="yes" />
    <input type="hidden" name="area" value="default" />
  </form>
</div>

          
        </div>

        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
          
            
            
              
            
            
              <ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../index.html">Introduction</a></li>
<li class="toctree-l1"><a class="reference internal" href="../installation_instructions.html">Installation instructions</a><ul>
<li class="toctree-l2"><a class="reference internal" href="../installation_instructions.html#general-installation">General installation</a></li>
<li class="toctree-l2"><a class="reference internal" href="../installation_instructions.html#installation-with-docker">Installation with Docker</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../how_to_run_tutorial.html">How to define and train models</a><ul>
<li class="toctree-l2"><a class="reference internal" href="../how_to_run_tutorial.html#arguments-for-launch-py">Arguments for launch.py</a></li>
<li class="toctree-l2"><a class="reference internal" href="../how_to_run_tutorial.html#arguments-for-run-py">Arguments for run.py</a></li>
<li class="toctree-l2"><a class="reference internal" href="../how_to_run_tutorial.html#configuration-file">Configuration file</a></li>
<li class="toctree-l2"><a class="reference internal" href="../how_to_run_tutorial.html#launching-jobs">Launching jobs</a></li>
</ul>
</li>
<li class="toctree-l1 current"><a class="reference internal" href="blocks.html">Tutorials and Recipes</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="getting_started.html">Getting started with building models in OpenChem</a><ul>
<li class="toctree-l3"><a class="reference internal" href="getting_started.html#loading-data">Loading data</a></li>
<li class="toctree-l3"><a class="reference internal" href="getting_started.html#creating-pytorch-dataset">Creating PyTorch dataset</a></li>
<li class="toctree-l3"><a class="reference internal" href="getting_started.html#creating-openchem-model-and-specifying-parameters">Creating OpenChem model and specifying parameters</a></li>
<li class="toctree-l3"><a class="reference internal" href="getting_started.html#training-the-model">Training the model</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="gcnn_tutorial.html">GraphCNN for predicting logP</a><ul>
<li class="toctree-l3"><a class="reference internal" href="gcnn_tutorial.html#defining-node-attributes">Defining node attributes</a></li>
<li class="toctree-l3"><a class="reference internal" href="gcnn_tutorial.html#loading-data">Loading data</a></li>
<li class="toctree-l3"><a class="reference internal" href="gcnn_tutorial.html#defining-model-architechture">Defining model architechture</a></li>
<li class="toctree-l3"><a class="reference internal" href="gcnn_tutorial.html#training-and-evaluating-the-model">Training and evaluating the model</a></li>
</ul>
</li>
<li class="toctree-l2 current"><a class="current reference internal" href="#">Tox21 Challenge</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#loading-data">Loading data</a></li>
<li class="toctree-l3"><a class="reference internal" href="#defining-evaluation-function">Defining evaluation function</a></li>
<li class="toctree-l3"><a class="reference internal" href="#defining-model-architechture">Defining model architechture</a></li>
<li class="toctree-l3"><a class="reference internal" href="#training-and-evaluating-the-model">Training and evaluating the model</a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../api-docs/blocks.html">API documentation</a><ul>
<li class="toctree-l2"><a class="reference internal" href="../api-docs/models.html">models</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../api-docs/models.html#module-models.openchem_model">openchem_model</a></li>
<li class="toctree-l3"><a class="reference internal" href="../api-docs/models.html#module-models.Smiles2Label">Smiles2Label</a></li>
<li class="toctree-l3"><a class="reference internal" href="../api-docs/models.html#module-models.Graph2Label">Graph2Label</a></li>
<li class="toctree-l3"><a class="reference internal" href="../api-docs/models.html#module-models.MoleculeProtein2Label">MoleculeProtein2Label</a></li>
<li class="toctree-l3"><a class="reference internal" href="../api-docs/models.html#vanilla-model">vanilla_model</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../api-docs/modules.html">modules</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../api-docs/modules.encoders.html">encoders</a><ul>
<li class="toctree-l4"><a class="reference internal" href="../api-docs/modules.encoders.html#module-modules.encoders.openchem_encoder">openchem_encoder</a></li>
<li class="toctree-l4"><a class="reference internal" href="../api-docs/modules.encoders.html#module-modules.encoders.rnn_encoder">rnn_encoder</a></li>
<li class="toctree-l4"><a class="reference internal" href="../api-docs/modules.encoders.html#cnn-encoder">cnn_encoder</a></li>
<li class="toctree-l4"><a class="reference internal" href="../api-docs/modules.encoders.html#gcnn-encoder">gcnn_encoder</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="../api-docs/modules.embeddings.html">embeddings</a><ul>
<li class="toctree-l4"><a class="reference internal" href="../api-docs/modules.embeddings.html#module-modules.embeddings.openchem_embedding">openchem_embedding</a></li>
<li class="toctree-l4"><a class="reference internal" href="../api-docs/modules.embeddings.html#module-modules.embeddings.basic_embedding">basic_embedding</a></li>
<li class="toctree-l4"><a class="reference internal" href="../api-docs/modules.embeddings.html#positional-embedding">positional_embedding</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="../api-docs/modules.mlp.html">mlp</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../api-docs/layers.html">layers</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../api-docs/layers.html#module-layers.conv_bn_relu">conv_bn_relu</a></li>
<li class="toctree-l3"><a class="reference internal" href="../api-docs/layers.html#module-layers.gcn">gcn</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../api-docs/data.html">data</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../api-docs/data.html#module-data.smiles_data_layer">smiles_data_layer</a></li>
<li class="toctree-l3"><a class="reference internal" href="../api-docs/data.html#graph-data-layer">graph_data_layer</a></li>
<li class="toctree-l3"><a class="reference internal" href="../api-docs/data.html#module-data.smiles_protein_data_layer">smiles_protein_data_layer</a></li>
<li class="toctree-l3"><a class="reference internal" href="../api-docs/data.html#module-data.vanilla_data_layer">vanilla_data_layer</a></li>
<li class="toctree-l3"><a class="reference internal" href="../api-docs/data.html#module-data.smiles_enumerator">smiles_enumerator</a></li>
<li class="toctree-l3"><a class="reference internal" href="../api-docs/data.html#module-data.utils">utils</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../api-docs/criterion.html">criterion</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../api-docs/criterion.html#module-openchem.criterion.multitask_loss">multitask_loss</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../api-docs/optimizer.html">optimizer</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../api-docs/optimizer.html#module-optimizer.openchem_optimizer">openchem_optimizer</a></li>
<li class="toctree-l3"><a class="reference internal" href="../api-docs/optimizer.html#module-optimizer.openchem_lr_scheduler">openchem_lr_scheduler</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../api-docs/utils.html">utils</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../api-docs/utils.html#module-openchem.utils.graph">graph</a></li>
<li class="toctree-l3"><a class="reference internal" href="../api-docs/utils.html#id1">utils</a></li>
<li class="toctree-l3"><a class="reference internal" href="../api-docs/utils.html#logger">logger</a></li>
</ul>
</li>
</ul>
</li>
</ul>

            
          
        </div>
      </div>
    </nav>

    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">

      
      <nav class="wy-nav-top" aria-label="top navigation">
        
          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
          <a href="../index.html">OpenChem</a>
        
      </nav>


      <div class="wy-nav-content">
        
        <div class="rst-content">
        
          

















<div role="navigation" aria-label="breadcrumbs navigation">

  <ul class="wy-breadcrumbs">
    
      <li><a href="../index.html">Docs</a> &raquo;</li>
        
          <li><a href="blocks.html">Tutorials and Recipes</a> &raquo;</li>
        
      <li>Tox21 Challenge</li>
    
    
      <li class="wy-breadcrumbs-aside">
        
            
            
              <!-- User defined GitHub URL -->
              <a href="https://github.com/Mariewelt/OpenChem" class="fa fa-github"> Edit on GitHub</a>
            
          
        
      </li>
    
  </ul>

  
  <hr/>
</div>
          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
           <div itemprop="articleBody">
            
  <div class="section" id="tox21-challenge">
<h1>Tox21 Challenge<a class="headerlink" href="#tox21-challenge" title="Permalink to this headline">¶</a></h1>
<p>In this tutorial we will build a Recurrent model for tox21 challenge.</p>
<div class="section" id="loading-data">
<h2>Loading data<a class="headerlink" href="#loading-data" title="Permalink to this headline">¶</a></h2>
<p>Tox21 dataset is available as a benchmark dataset, so you can load it from benchmark datasets folder with OpenChem <code class="docutils literal notranslate"><span class="pre">read_smiles_property_file</span></code> function:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="kn">from</span> <span class="nn">openchem.data.utils</span> <span class="n">load</span> <span class="n">read_smiles_property_file</span>
<span class="n">data</span> <span class="o">=</span> <span class="n">read_smiles_property_file</span><span class="p">(</span><span class="s1">&#39;./benchmark_datasets/tox21/tox21.csv&#39;</span><span class="p">,</span>
                                 <span class="n">cols_to_read</span><span class="o">=</span><span class="p">[</span><span class="mi">13</span><span class="p">]</span> <span class="o">+</span> <span class="nb">list</span><span class="p">(</span><span class="nb">range</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span><span class="mi">12</span><span class="p">)))</span>
<span class="n">smiles</span> <span class="o">=</span> <span class="n">data</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="n">labels</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="mi">1</span><span class="p">:])</span>
</pre></div>
</div>
<p>Tox21 data requires some preprocessing. As it is a multi-target dataset, some of the labels are not available and therefore just left empty. We need to fill them with dummy index, that will be ignored during training. Let’s choose ‘999’ as a dummy index:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">labels</span><span class="p">[</span><span class="n">np</span><span class="o">.</span><span class="n">where</span><span class="p">(</span><span class="n">labels</span><span class="o">==</span><span class="s1">&#39;&#39;</span><span class="p">)]</span> <span class="o">=</span> <span class="s1">&#39;999&#39;</span>
<span class="n">labels</span> <span class="o">=</span> <span class="n">labels</span><span class="o">.</span><span class="n">T</span>
</pre></div>
</div>
<p>We will also extract unique tokens from the whole dataset before splitting it into train and test in order to avoid the situation, when some of the tokens will not be present in one of the pieces of the dataset:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">openchem.data.utils</span> <span class="kn">import</span> <span class="n">get_tokens</span>
<span class="n">tokens</span><span class="p">,</span> <span class="n">_</span><span class="p">,</span> <span class="n">_</span> <span class="o">=</span> <span class="n">get_tokens</span><span class="p">(</span><span class="n">smiles</span><span class="p">)</span>
<span class="n">tokens</span> <span class="o">=</span> <span class="n">tokens</span> <span class="o">+</span> <span class="s1">&#39; &#39;</span>
</pre></div>
</div>
<p>Now we will split data into training and test:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">sklearn.model_selection</span> <span class="kn">import</span> <span class="n">train_test_split</span>
<span class="n">X_train</span><span class="p">,</span> <span class="n">X_test</span><span class="p">,</span> <span class="n">y_train</span><span class="p">,</span> <span class="n">y_test</span> <span class="o">=</span> <span class="n">train_test_split</span><span class="p">(</span><span class="n">smiles</span><span class="p">,</span> <span class="n">labels</span><span class="p">,</span> <span class="n">test_size</span><span class="o">=</span><span class="mf">0.2</span><span class="p">,</span>
                                                    <span class="n">random_state</span><span class="o">=</span><span class="mi">42</span><span class="p">)</span>
</pre></div>
</div>
<p>And save train and test splits to new files with OpenChem <code class="docutils literal notranslate"><span class="pre">save_smiles_property_file</span></code> utility:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">openchem.data.utils</span> <span class="kn">import</span> <span class="n">save_smiles_property_file</span>
<span class="n">save_smiles_property_file</span><span class="p">(</span><span class="s1">&#39;./benchmark_datasets/tox21/train.smi&#39;</span><span class="p">,</span> <span class="n">X_train</span><span class="p">,</span> <span class="n">y_train</span><span class="p">)</span>
<span class="n">save_smiles_property_file</span><span class="p">(</span><span class="s1">&#39;./benchmark_datasets/tox21/test.smi&#39;</span><span class="p">,</span> <span class="n">X_test</span><span class="p">,</span> <span class="n">y_test</span><span class="p">)</span>
</pre></div>
</div>
<p>Now you can create SMILES data layer from input files. We will pass tokens as an argument for data layer. We will also use data augmentation by SMILES <a class="reference external" href="https://arxiv.org/abs/1703.07076">enumeration</a>. The idea behind it is to include non-canonical notation for SMILES. Augmentation is enabled by setting the argument <code class="docutils literal notranslate"><span class="pre">augment=True</span></code> when creating an object of class <code class="xref py py-class docutils literal notranslate"><span class="pre">SmilesDataset</span></code>:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">openchem.data.graph_data_layer</span> <span class="kn">import</span> <span class="n">SmilesDataset</span>
<span class="n">train_dataset</span> <span class="o">=</span> <span class="n">SmilesDataset</span><span class="p">(</span><span class="s1">&#39;./benchmark_datasets/tox21/train.smi&#39;</span><span class="p">,</span>
                              <span class="n">delimiter</span><span class="o">=</span><span class="s1">&#39;,&#39;</span><span class="p">,</span> <span class="n">cols_to_read</span><span class="o">=</span><span class="nb">list</span><span class="p">(</span><span class="nb">range</span><span class="p">(</span><span class="mi">13</span><span class="p">)),</span>
                              <span class="n">tokens</span><span class="o">=</span><span class="n">tokens</span><span class="p">,</span> <span class="n">augment</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
<span class="n">test_dataset</span> <span class="o">=</span> <span class="n">SmilesDataset</span><span class="p">(</span><span class="s1">&#39;./benchmark_datasets/tox21/test.smi&#39;</span><span class="p">,</span>
                            <span class="n">delimiter</span><span class="o">=</span><span class="s1">&#39;,&#39;</span><span class="p">,</span> <span class="n">cols_to_read</span><span class="o">=</span><span class="nb">list</span><span class="p">(</span><span class="nb">range</span><span class="p">(</span><span class="mi">13</span><span class="p">)),</span>
                            <span class="n">tokens</span><span class="o">=</span><span class="n">tokens</span><span class="p">)</span>
</pre></div>
</div>
<p>Note that we only need to augment training dataset.</p>
</div>
<div class="section" id="defining-evaluation-function">
<h2>Defining evaluation function<a class="headerlink" href="#defining-evaluation-function" title="Permalink to this headline">¶</a></h2>
<p>We will also need to implement our own evaluation function for calculating classification accuracy separately for each task. As an accuracy metrics we will use AUC:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">multitask_auc</span><span class="p">(</span><span class="n">ground_truth</span><span class="p">,</span> <span class="n">predicted</span><span class="p">):</span>
<span class="kn">from</span> <span class="nn">sklearn.metrics</span> <span class="kn">import</span> <span class="n">roc_auc_score</span>
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="n">ground_truth</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">(</span><span class="n">ground_truth</span><span class="p">)</span>
<span class="n">predicted</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">(</span><span class="n">predicted</span><span class="p">)</span>
<span class="n">n_tasks</span> <span class="o">=</span> <span class="n">ground_truth</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
<span class="n">auc</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">n_tasks</span><span class="p">):</span>
    <span class="n">ind</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">where</span><span class="p">(</span><span class="n">ground_truth</span><span class="p">[:,</span> <span class="n">i</span><span class="p">]</span> <span class="o">!=</span> <span class="mi">999</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span>
    <span class="n">auc</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">roc_auc_score</span><span class="p">(</span><span class="n">ground_truth</span><span class="p">[</span><span class="n">ind</span><span class="p">,</span> <span class="n">i</span><span class="p">],</span> <span class="n">predicted</span><span class="p">[</span><span class="n">ind</span><span class="p">,</span> <span class="n">i</span><span class="p">]))</span>
<span class="k">return</span> <span class="n">np</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span><span class="n">auc</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="section" id="defining-model-architechture">
<h2>Defining model architechture<a class="headerlink" href="#defining-model-architechture" title="Permalink to this headline">¶</a></h2>
<p>Now we define model architecture. We will use <code class="xref py py-class docutils literal notranslate"><span class="pre">Smiles2Label</span></code> modality.</p>
<p>This model consists of Embedding block, Recurrent Encoder with 4 LSTM layers and MLP. We will use dropout with high probability to enable regularization to avoid model overfitting:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">model</span> <span class="o">=</span> <span class="n">Smiles2Label</span>

<span class="n">model_params</span> <span class="o">=</span> <span class="p">{</span>
    <span class="s1">&#39;use_cuda&#39;</span><span class="p">:</span> <span class="kc">True</span><span class="p">,</span>
    <span class="s1">&#39;task&#39;</span><span class="p">:</span> <span class="s1">&#39;multitask&#39;</span><span class="p">,</span>
    <span class="s1">&#39;random_seed&#39;</span><span class="p">:</span> <span class="mi">5</span><span class="p">,</span>
    <span class="s1">&#39;use_clip_grad&#39;</span><span class="p">:</span> <span class="kc">True</span><span class="p">,</span>
    <span class="s1">&#39;max_grad_norm&#39;</span><span class="p">:</span> <span class="mf">10.0</span><span class="p">,</span>
    <span class="s1">&#39;batch_size&#39;</span><span class="p">:</span> <span class="mi">256</span><span class="p">,</span>
    <span class="s1">&#39;num_epochs&#39;</span><span class="p">:</span> <span class="mi">21</span><span class="p">,</span>
    <span class="s1">&#39;logdir&#39;</span><span class="p">:</span> <span class="s1">&#39;./logs/tox21_rnn_log&#39;</span><span class="p">,</span>
    <span class="s1">&#39;print_every&#39;</span><span class="p">:</span> <span class="mi">5</span><span class="p">,</span>
    <span class="s1">&#39;save_every&#39;</span><span class="p">:</span> <span class="mi">5</span><span class="p">,</span>
    <span class="s1">&#39;train_data_layer&#39;</span><span class="p">:</span> <span class="n">train_dataset</span><span class="p">,</span>
    <span class="s1">&#39;val_data_layer&#39;</span><span class="p">:</span> <span class="n">test_dataset</span><span class="p">,</span>
    <span class="s1">&#39;predict_data_layer&#39;</span><span class="p">:</span> <span class="n">predict_dataset</span><span class="p">,</span>
    <span class="s1">&#39;eval_metrics&#39;</span><span class="p">:</span> <span class="n">multitask_auc</span><span class="p">,</span>
    <span class="s1">&#39;criterion&#39;</span><span class="p">:</span> <span class="n">MultitaskLoss</span><span class="p">(</span><span class="n">ignore_index</span><span class="o">=</span><span class="mi">9</span><span class="p">,</span> <span class="n">n_tasks</span><span class="o">=</span><span class="mi">12</span><span class="p">)</span><span class="o">.</span><span class="n">cuda</span><span class="p">(),</span>
    <span class="s1">&#39;optimizer&#39;</span><span class="p">:</span> <span class="n">RMSprop</span><span class="p">,</span>
    <span class="s1">&#39;optimizer_params&#39;</span><span class="p">:</span> <span class="p">{</span>
        <span class="s1">&#39;lr&#39;</span><span class="p">:</span> <span class="mf">0.001</span><span class="p">,</span>
        <span class="p">},</span>
    <span class="s1">&#39;lr_scheduler&#39;</span><span class="p">:</span> <span class="n">StepLR</span><span class="p">,</span>
    <span class="s1">&#39;lr_scheduler_params&#39;</span><span class="p">:</span> <span class="p">{</span>
        <span class="s1">&#39;step_size&#39;</span><span class="p">:</span> <span class="mi">10</span><span class="p">,</span>
        <span class="s1">&#39;gamma&#39;</span><span class="p">:</span> <span class="mf">0.8</span>
    <span class="p">},</span>
    <span class="s1">&#39;embedding&#39;</span><span class="p">:</span> <span class="n">Embedding</span><span class="p">,</span>
    <span class="s1">&#39;embedding_params&#39;</span><span class="p">:</span> <span class="p">{</span>
        <span class="s1">&#39;num_embeddings&#39;</span><span class="p">:</span> <span class="n">train_dataset</span><span class="o">.</span><span class="n">num_tokens</span><span class="p">,</span>
        <span class="s1">&#39;embedding_dim&#39;</span><span class="p">:</span> <span class="mi">128</span><span class="p">,</span>
        <span class="s1">&#39;padding_idx&#39;</span><span class="p">:</span> <span class="n">train_dataset</span><span class="o">.</span><span class="n">tokens</span><span class="o">.</span><span class="n">index</span><span class="p">(</span><span class="s1">&#39; &#39;</span><span class="p">)</span>
    <span class="p">},</span>
    <span class="s1">&#39;encoder&#39;</span><span class="p">:</span> <span class="n">RNNEncoder</span><span class="p">,</span>
    <span class="s1">&#39;encoder_params&#39;</span><span class="p">:</span> <span class="p">{</span>
        <span class="s1">&#39;input_size&#39;</span><span class="p">:</span> <span class="mi">128</span><span class="p">,</span>
        <span class="s1">&#39;layer&#39;</span><span class="p">:</span> <span class="s2">&quot;LSTM&quot;</span><span class="p">,</span>
        <span class="s1">&#39;encoder_dim&#39;</span><span class="p">:</span> <span class="mi">128</span><span class="p">,</span>
        <span class="s1">&#39;n_layers&#39;</span><span class="p">:</span> <span class="mi">4</span><span class="p">,</span>
        <span class="s1">&#39;dropout&#39;</span><span class="p">:</span> <span class="mf">0.8</span><span class="p">,</span>
        <span class="s1">&#39;is_bidirectional&#39;</span><span class="p">:</span> <span class="kc">False</span>
    <span class="p">},</span>
    <span class="s1">&#39;mlp&#39;</span><span class="p">:</span> <span class="n">OpenChemMLP</span><span class="p">,</span>
    <span class="s1">&#39;mlp_params&#39;</span><span class="p">:</span> <span class="p">{</span>
        <span class="s1">&#39;input_size&#39;</span><span class="p">:</span> <span class="mi">128</span><span class="p">,</span>
        <span class="s1">&#39;n_layers&#39;</span><span class="p">:</span> <span class="mi">2</span><span class="p">,</span>
        <span class="s1">&#39;hidden_size&#39;</span><span class="p">:</span> <span class="p">[</span><span class="mi">128</span><span class="p">,</span> <span class="mi">12</span><span class="p">],</span>
        <span class="s1">&#39;activation&#39;</span><span class="p">:</span> <span class="p">[</span><span class="n">F</span><span class="o">.</span><span class="n">relu</span><span class="p">,</span> <span class="n">torch</span><span class="o">.</span><span class="n">sigmoid</span><span class="p">],</span>
        <span class="s1">&#39;dropout&#39;</span><span class="p">:</span> <span class="mf">0.0</span>
    <span class="p">}</span>
<span class="p">}</span>
</pre></div>
</div>
<p>All of the above code should be saved in a python file. We will call it <code class="docutils literal notranslate"><span class="pre">tox21_rnn_config.py</span></code>.</p>
</div>
<div class="section" id="training-and-evaluating-the-model">
<h2>Training and evaluating the model<a class="headerlink" href="#training-and-evaluating-the-model" title="Permalink to this headline">¶</a></h2>
<p>Now as we loaded the datasets and defined the model architechture we can launch training and evaluation process from the terminal.</p>
<p>Suppose we have a machine with 4 GPUs, so we want to run training in distributed mode. We also want to see the evaluation metrics while the training is in progress. All the parameters from config file can be redefined in command line and parsed to the run script as arguments. So, we can, for example, change the batch size and number of epochs:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">python</span> <span class="n">launch</span><span class="o">.</span><span class="n">py</span> <span class="o">--</span><span class="n">nproc_per_node</span><span class="o">=</span><span class="mi">4</span> <span class="n">run</span><span class="o">.</span><span class="n">py</span> <span class="o">--</span><span class="n">config_file</span><span class="o">=</span><span class="s2">&quot;./tox21_rnn_config.py&quot;</span> <span class="o">--</span><span class="n">mode</span><span class="o">=</span><span class="s2">&quot;train_eval&quot;</span> <span class="o">--</span><span class="n">batch_size</span><span class="o">=</span><span class="mi">256</span> <span class="o">--</span><span class="n">num_epochs</span><span class="o">=</span><span class="mi">50</span>
</pre></div>
</div>
<p>The output will be:</p>
<p>Model checkpoints and tensorboard log are saved to <code class="docutils literal notranslate"><span class="pre">logdir</span></code> folder specified in the configuration file.</p>
<p>Now you can evaluate model:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">python</span> <span class="n">launch</span><span class="o">.</span><span class="n">py</span> <span class="o">--</span><span class="n">nproc_per_node</span><span class="o">=</span><span class="mi">4</span> <span class="n">run</span><span class="o">.</span><span class="n">py</span> <span class="o">--</span><span class="n">config_file</span><span class="o">=</span><span class="s2">&quot;./tox21_rnn_config.py&quot;</span> <span class="o">--</span><span class="n">mode</span><span class="o">=</span><span class="s2">&quot;eval&quot;</span>
</pre></div>
</div>
<p>The output will be:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="o">***</span> <span class="n">Starting</span> <span class="n">training</span> <span class="kn">from</span> <span class="nn">scratch</span> <span class="n">process</span> <span class="mi">3</span>
<span class="o">***</span> <span class="n">Starting</span> <span class="n">training</span> <span class="kn">from</span> <span class="nn">scratch</span> <span class="n">process</span> <span class="mi">0</span>
<span class="o">***</span> <span class="n">Starting</span> <span class="n">training</span> <span class="kn">from</span> <span class="nn">scratch</span> <span class="n">process</span> <span class="mi">1</span>
<span class="o">***</span> <span class="n">Starting</span> <span class="n">training</span> <span class="kn">from</span> <span class="nn">scratch</span> <span class="n">process</span> <span class="mi">2</span>
<span class="n">Distributed</span> <span class="n">process</span> <span class="k">with</span> <span class="n">rank</span> <span class="mi">1</span> <span class="n">initiated</span>
<span class="n">Distributed</span> <span class="n">process</span> <span class="k">with</span> <span class="n">rank</span> <span class="mi">2</span> <span class="n">initiated</span>
<span class="n">Distributed</span> <span class="n">process</span> <span class="k">with</span> <span class="n">rank</span> <span class="mi">0</span> <span class="n">initiated</span>
<span class="n">Distributed</span> <span class="n">process</span> <span class="k">with</span> <span class="n">rank</span> <span class="mi">3</span> <span class="n">initiated</span>
<span class="n">TRAINING</span><span class="p">:</span> <span class="p">[</span><span class="n">Time</span><span class="p">:</span> <span class="mi">0</span><span class="n">m</span> <span class="mi">13</span><span class="n">s</span><span class="p">,</span> <span class="n">Epoch</span><span class="p">:</span> <span class="mi">0</span><span class="p">,</span> <span class="n">Progress</span><span class="p">:</span> <span class="mi">0</span><span class="o">%</span><span class="p">,</span> <span class="n">Loss</span><span class="p">:</span> <span class="mf">0.3052</span><span class="p">]</span>
<span class="n">EVALUATION</span><span class="p">:</span> <span class="p">[</span><span class="n">Time</span><span class="p">:</span> <span class="mi">0</span><span class="n">m</span> <span class="mi">0</span><span class="n">s</span><span class="p">,</span> <span class="n">Loss</span><span class="p">:</span> <span class="mf">0.3071</span><span class="p">,</span> <span class="n">Metrics</span><span class="p">:</span> <span class="mf">0.6030</span><span class="p">]</span>
<span class="n">TRAINING</span><span class="p">:</span> <span class="p">[</span><span class="n">Time</span><span class="p">:</span> <span class="mi">1</span><span class="n">m</span> <span class="mi">18</span><span class="n">s</span><span class="p">,</span> <span class="n">Epoch</span><span class="p">:</span> <span class="mi">5</span><span class="p">,</span> <span class="n">Progress</span><span class="p">:</span> <span class="mi">16</span><span class="o">%</span><span class="p">,</span> <span class="n">Loss</span><span class="p">:</span> <span class="mf">0.1932</span><span class="p">]</span>
<span class="n">EVALUATION</span><span class="p">:</span> <span class="p">[</span><span class="n">Time</span><span class="p">:</span> <span class="mi">0</span><span class="n">m</span> <span class="mi">0</span><span class="n">s</span><span class="p">,</span> <span class="n">Loss</span><span class="p">:</span> <span class="mf">0.1867</span><span class="p">,</span> <span class="n">Metrics</span><span class="p">:</span> <span class="mf">0.7948</span><span class="p">]</span>
<span class="n">TRAINING</span><span class="p">:</span> <span class="p">[</span><span class="n">Time</span><span class="p">:</span> <span class="mi">2</span><span class="n">m</span> <span class="mi">24</span><span class="n">s</span><span class="p">,</span> <span class="n">Epoch</span><span class="p">:</span> <span class="mi">10</span><span class="p">,</span> <span class="n">Progress</span><span class="p">:</span> <span class="mi">32</span><span class="o">%</span><span class="p">,</span> <span class="n">Loss</span><span class="p">:</span> <span class="mf">0.1828</span><span class="p">]</span>
<span class="n">EVALUATION</span><span class="p">:</span> <span class="p">[</span><span class="n">Time</span><span class="p">:</span> <span class="mi">0</span><span class="n">m</span> <span class="mi">0</span><span class="n">s</span><span class="p">,</span> <span class="n">Loss</span><span class="p">:</span> <span class="mf">0.1807</span><span class="p">,</span> <span class="n">Metrics</span><span class="p">:</span> <span class="mf">0.8187</span><span class="p">]</span>
<span class="n">TRAINING</span><span class="p">:</span> <span class="p">[</span><span class="n">Time</span><span class="p">:</span> <span class="mi">3</span><span class="n">m</span> <span class="mi">30</span><span class="n">s</span><span class="p">,</span> <span class="n">Epoch</span><span class="p">:</span> <span class="mi">15</span><span class="p">,</span> <span class="n">Progress</span><span class="p">:</span> <span class="mi">48</span><span class="o">%</span><span class="p">,</span> <span class="n">Loss</span><span class="p">:</span> <span class="mf">0.1733</span><span class="p">]</span>
<span class="n">EVALUATION</span><span class="p">:</span> <span class="p">[</span><span class="n">Time</span><span class="p">:</span> <span class="mi">0</span><span class="n">m</span> <span class="mi">0</span><span class="n">s</span><span class="p">,</span> <span class="n">Loss</span><span class="p">:</span> <span class="mf">0.1794</span><span class="p">,</span> <span class="n">Metrics</span><span class="p">:</span> <span class="mf">0.8296</span><span class="p">]</span>
<span class="n">TRAINING</span><span class="p">:</span> <span class="p">[</span><span class="n">Time</span><span class="p">:</span> <span class="mi">4</span><span class="n">m</span> <span class="mi">36</span><span class="n">s</span><span class="p">,</span> <span class="n">Epoch</span><span class="p">:</span> <span class="mi">20</span><span class="p">,</span> <span class="n">Progress</span><span class="p">:</span> <span class="mi">64</span><span class="o">%</span><span class="p">,</span> <span class="n">Loss</span><span class="p">:</span> <span class="mf">0.1680</span><span class="p">]</span>
<span class="n">EVALUATION</span><span class="p">:</span> <span class="p">[</span><span class="n">Time</span><span class="p">:</span> <span class="mi">0</span><span class="n">m</span> <span class="mi">0</span><span class="n">s</span><span class="p">,</span> <span class="n">Loss</span><span class="p">:</span> <span class="mf">0.1766</span><span class="p">,</span> <span class="n">Metrics</span><span class="p">:</span> <span class="mf">0.8380</span><span class="p">]</span>
<span class="n">TRAINING</span><span class="p">:</span> <span class="p">[</span><span class="n">Time</span><span class="p">:</span> <span class="mi">5</span><span class="n">m</span> <span class="mi">43</span><span class="n">s</span><span class="p">,</span> <span class="n">Epoch</span><span class="p">:</span> <span class="mi">25</span><span class="p">,</span> <span class="n">Progress</span><span class="p">:</span> <span class="mi">80</span><span class="o">%</span><span class="p">,</span> <span class="n">Loss</span><span class="p">:</span> <span class="mf">0.1637</span><span class="p">]</span>
<span class="n">EVALUATION</span><span class="p">:</span> <span class="p">[</span><span class="n">Time</span><span class="p">:</span> <span class="mi">0</span><span class="n">m</span> <span class="mi">0</span><span class="n">s</span><span class="p">,</span> <span class="n">Loss</span><span class="p">:</span> <span class="mf">0.1778</span><span class="p">,</span> <span class="n">Metrics</span><span class="p">:</span> <span class="mf">0.8352</span><span class="p">]</span>
<span class="n">TRAINING</span><span class="p">:</span> <span class="p">[</span><span class="n">Time</span><span class="p">:</span> <span class="mi">6</span><span class="n">m</span> <span class="mi">48</span><span class="n">s</span><span class="p">,</span> <span class="n">Epoch</span><span class="p">:</span> <span class="mi">30</span><span class="p">,</span> <span class="n">Progress</span><span class="p">:</span> <span class="mi">96</span><span class="o">%</span><span class="p">,</span> <span class="n">Loss</span><span class="p">:</span> <span class="mf">0.1614</span><span class="p">]</span>
<span class="n">EVALUATION</span><span class="p">:</span> <span class="p">[</span><span class="n">Time</span><span class="p">:</span> <span class="mi">0</span><span class="n">m</span> <span class="mi">0</span><span class="n">s</span><span class="p">,</span> <span class="n">Loss</span><span class="p">:</span> <span class="mf">0.1763</span><span class="p">,</span> <span class="n">Metrics</span><span class="p">:</span> <span class="mf">0.8379</span><span class="p">]</span>
</pre></div>
</div>
<p>Next you can run evalutaion:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">python</span> <span class="n">launch</span><span class="o">.</span><span class="n">py</span> <span class="o">--</span><span class="n">nproc_per_node</span><span class="o">=</span><span class="mi">4</span> <span class="n">run</span><span class="o">.</span><span class="n">py</span> <span class="o">--</span><span class="n">config_file</span><span class="o">=</span><span class="s2">&quot;./tox21_rnn_config.py&quot;</span> <span class="o">--</span><span class="n">mode</span><span class="o">=</span><span class="s2">&quot;eval&quot;</span>

<span class="o">***</span> <span class="n">Loading</span> <span class="n">model</span> <span class="kn">from</span> <span class="o">/</span><span class="n">home</span><span class="o">/</span><span class="n">user</span><span class="o">/</span><span class="n">OpenChem</span><span class="o">/</span><span class="n">logs</span><span class="o">/</span><span class="n">checkpoint</span><span class="o">/</span><span class="n">epoch_30</span>
<span class="o">***</span> <span class="n">Loading</span> <span class="n">model</span> <span class="kn">from</span> <span class="o">/</span><span class="n">home</span><span class="o">/</span><span class="n">user</span><span class="o">/</span><span class="n">OpenChem</span><span class="o">/</span><span class="n">logs</span><span class="o">/</span><span class="n">checkpoint</span><span class="o">/</span><span class="n">epoch_30</span>
<span class="o">***</span> <span class="n">Loading</span> <span class="n">model</span> <span class="kn">from</span> <span class="o">/</span><span class="n">home</span><span class="o">/</span><span class="n">user</span><span class="o">/</span><span class="n">OpenChem</span><span class="o">/</span><span class="n">logs</span><span class="o">/</span><span class="n">checkpoint</span><span class="o">/</span><span class="n">epoch_30</span>
<span class="o">***</span> <span class="n">Loading</span> <span class="n">model</span> <span class="kn">from</span> <span class="o">/</span><span class="n">home</span><span class="o">/</span><span class="n">user</span><span class="o">/</span><span class="n">OpenChem</span><span class="o">/</span><span class="n">logs</span><span class="o">/</span><span class="n">checkpoint</span><span class="o">/</span><span class="n">epoch_30</span>
<span class="n">Distributed</span> <span class="n">process</span> <span class="k">with</span> <span class="n">rank</span> <span class="mi">3</span> <span class="n">initiated</span>
<span class="n">Distributed</span> <span class="n">process</span> <span class="k">with</span> <span class="n">rank</span> <span class="mi">1</span> <span class="n">initiated</span>
<span class="n">Distributed</span> <span class="n">process</span> <span class="k">with</span> <span class="n">rank</span> <span class="mi">2</span> <span class="n">initiated</span>
<span class="n">Distributed</span> <span class="n">process</span> <span class="k">with</span> <span class="n">rank</span> <span class="mi">0</span> <span class="n">initiated</span>
<span class="o">=&gt;</span> <span class="n">loading</span> <span class="n">model</span>  <span class="n">pre</span><span class="o">-</span><span class="n">trained</span> <span class="n">model</span>
<span class="o">=&gt;</span> <span class="n">loading</span> <span class="n">model</span>  <span class="n">pre</span><span class="o">-</span><span class="n">trained</span> <span class="n">model</span>
<span class="o">=&gt;</span> <span class="n">loading</span> <span class="n">model</span>  <span class="n">pre</span><span class="o">-</span><span class="n">trained</span> <span class="n">model</span>
<span class="o">=&gt;</span> <span class="n">loading</span> <span class="n">model</span>  <span class="n">pre</span><span class="o">-</span><span class="n">trained</span> <span class="n">model</span>
<span class="n">EVALUATION</span><span class="p">:</span> <span class="p">[</span><span class="n">Time</span><span class="p">:</span> <span class="mi">0</span><span class="n">m</span> <span class="mi">0</span><span class="n">s</span><span class="p">,</span> <span class="n">Loss</span><span class="p">:</span> <span class="mf">0.1763</span><span class="p">,</span> <span class="n">Metrics</span><span class="p">:</span> <span class="mf">0.8379</span><span class="p">]</span>
</pre></div>
</div>
<p>So, we trained a Multi-task Recurrent Neural Network for predicting biological activity for 12 receptors from tox21 challenge with mean AUC of ~0.84.</p>
<p>If we want to calculate per target AUC, we will need to change the external metrics function a little bit – for example, by just adding the print statement to print per target AUCs. So, with this model we obtain the following per target AUCs on test set:</p>
<ul class="simple">
<li><p>NR-AR 0.85</p></li>
<li><p>NR-AR-LBD 0.90</p></li>
<li><p>NR-AhR 0.87</p></li>
<li><p>NR-Aromatase 0.84</p></li>
<li><p>NR-ER 0.76</p></li>
<li><p>NR-ER-LBD 0.82</p></li>
<li><p>NR-PPAR-gamma 0.80</p></li>
<li><p>SR-ARE 0.78</p></li>
<li><p>SR-ATAD5 0.85</p></li>
<li><p>SR-HSE 0.84</p></li>
<li><p>SR-MMP 0.87</p></li>
<li><p>SR-p53 0.86</p></li>
</ul>
</div>
</div>


           </div>
           
          </div>
          <footer>
  
    <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
      
        <a href="../api-docs/blocks.html" class="btn btn-neutral float-right" title="API documentation" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right"></span></a>
      
      
        <a href="gcnn_tutorial.html" class="btn btn-neutral float-left" title="GraphCNN for predicting logP" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left"></span> Previous</a>
      
    </div>
  

  <hr/>

  <div role="contentinfo">
    <p>
        &copy; Copyright 2018, Mariya Popova

    </p>
  </div>
  Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/rtfd/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>. 

</footer>

        </div>
      </div>

    </section>

  </div>
  


  <script type="text/javascript">
      jQuery(function () {
          SphinxRtdTheme.Navigation.enable(true);
      });
  </script>

  
  
    
    

  <style>
    /* Sidebar header (and topbar for mobile) */
    .wy-side-nav-search, .wy-nav-top {
      background: #99badd;
    }
  </style>


</body>
</html>